Method for scrubbing regions in central storage

ABSTRACT

Memory is scrubbed by an improved non-linear method giving scrubbing preference to the central storage region having the characteristic of a high risk read-only memory such as the CPA region to prevent the accumulation of temporary data errors. The chip row on which the CPA resides is scrubbed after each time the scrubbing of a non-CPA chip row in a PMA completed successfully. The next non-CPA least recently scrubbed chip row would be selected for scrubbing after scrubbing completed on the CPA chip row. This in a first case provides non-linear selection methods of scrubbing central storage of computer systems to more frequently select (“select” herein encompasses the meaning of “favor”) scrub regions having the characteristic of a predominately read-only memory making those regions at a higher risk of failure than those regions having lower risk because of frequent write operations. In a second case, scrub regions having the characteristic of a predominately read-only memory are selected by using a second preferred embodiment selection method which uses the detection of faulty data from normal system accesses to central storage to identify other high risk regions and scrub them before other lower risk regions. In addition, the severity of the detected data error can be used to determine the rate at which scrub commands are sent to the selected region: the higher the severity, the higher the scrub rate.

TRADEMARKS

IBM® is a registered trademark of International Business MachinesCorporation, Armonk, N.Y., U.S.A. S/390, Z900 and z990 and other namesused herein may be registered trademarks, trademarks or product names ofInternational Business Machines Corporation or other companies.

FIELD OF THE INVENTION

This invention is related to computers and computer systems and inparticular to methods to scrubbing a high risk region of central storageto be processed for the removal of temporary errors by scrubbing theregion using different scrub rates based on a risk assessment.

BACKGROUND OF THE INVENTION

Scrubbing main memory is a practice used in IBM, as in the z900 Seriessystems illustrated by U.S. Pat. No. 6,446,145 issued Sep. 3, 2002illustrating linear scrubbing in the prior art.

From prior art, each of the DRAM chip row regions, comprising aProcessor Memory Array (PMA) of one or more PMAs comprising a centralstorage, is selected for scrubbing in a linear fashion. That is, afterchip row n is scrubbed, chip row n+1 is selected for scrubbing and afterthe last chip row is scrubbed, chip row 0 is again selected.

The scrub process begins by fetching a unit of data containing ECC wordsfrom central storage, the detection of a single bit error (CE) withinthe ECC word or single symbol error (two bit error from the sameDRAM—also a CE) within an ECC word, the absence of multi-bit errors (twoor more bit errors that span more than one symbol—UE) within any ECCword within the unit of data, and the store back of the unit of datawith the temporary single bit errors or single symbol errors beingcorrected by the ECC correction circuitry. The region of central storagebeing defined as the space occupied by the ECC words contained in onerow of DRAM chips.

Background scrubbing of memory cards on z900 servers is under millicodecontrol. Every millisecond the millicode issues 8 separate operations toscrub 256 bytes per operation. It takes approximately 9.32 hours toscrub 64 Gigabytes(GB) of memory.

The operating system control program, which contains a greaterpercentage of read-only regions than customer storage, and which residescontiguously in the low 2 GB of storage for z900 servers, is a high riskregion. In a 9.32 hour time frame, the control program area (CPA) isonly scrubbed once. If the CPA memory contained temporary errors(Correctable Errors—CE's), these errors may not be corrected by storesto those read-only CPA locations. These read-only regions then dependtotally on scrubbing to correct the possible temporary CE's. The concernis that these CE's may not be corrected before another CE appears in thesame ECC word to result in an Uncorrectable Error (UE), and an UE in CPAis a system check-stop event.

When millicode completes scrubbing a memory chip row, or rank, millicodeexamines Bit Error Counters for a threshold condition (a condition wherea Bit Error Counter equals or exceeds a predetermined value). There isone Bit

Error Counter for each DRAM in a chip row. The same set of counters isshared by all chip rows, since each chip row is scrubbed separately. Ifa DRAM on that chip row has its corresponding Bit Error Counter reachthe threshold condition, then millicode would attempt to replace thisDRAM with a spare DRAM. The attempt is successful if the spare DRAM isnot already in use and the spare DRAM is in good condition: its BitError Counter did not reach threshold. At this time, the memory accessto that DRAM is put into Half-Spare mode. This means that the stores tothe bad DRAM will also be stored to the spare DRAM and the fetches tothe bad DRAM will still only come from the bad DRAM. When scrubbing isperformed again for this chip row, the data in the bad DRAM would bemoved over to the spare DRAM. At the end of re-scrubbing this chip row,the memory accesses to the bad DRAM will be switched to Full-Spare modeby millicode. All fetches will now come from the spare DRAM.

For z990 servers with linear scrub region selection as in prior art, itis desired that all of memory in central storage, a possible maximum of128G per book for a maximum of 4 books, be scrubbed once within an8-hour shift. This is to be achieved by a combination of using the z990server's scrub command which scrubs up to 1024 bytes per PMA peroperation, and sending 4 operations every 250 microseconds. The CPA areawould be scrubbed once in 8.68 hours.

SUMMARY OF THE INVENTION

By employing the inventions described below, we have learned here at IBMthat using risk assessment to select a scrub region and a scrub rate,the probability increases that temporary faulty data residing in centralstorage can be corrected before potential damage is caused to the systemor that components with permanent damage can be electronically replaced(spared) and that the scrubbing methods described herein are capable ofmeeting objectives.

A first preferred embodiment of our invention provides non-linearselection methods of scrubbing central storage of computer systems tomore frequently select (“select” herein encompasses the meaning of“favor”) scrub regions having the characteristic of a predominatelyread-only memory making those regions at a higher risk of failure thanthose regions having lower risk because of frequent write operations.

In accordance with our invention, the first preferred method ofselection is enhanced to modify the non-linear selection methodselecting scrub regions having the characteristic of a predominatelyread-only memory, by using a second preferred embodiment selectionmethod which uses the detection of faulty data from normal systemaccesses to central storage to identify other high risk regions andscrub them before other lower risk regions. Sometimes, in the secondpreferred embodiment of the method, in addition, the severity of thedetected data error would be used determine the rate at which scrubcommands are sent to the selected region: the higher the severity, thehigher the scrub rate.

For a fuller understanding of the inventions, reference should be madeto the detailed description which follows.

TABLES AND DRAWINGS ILLUSTRATING THE INVENTION

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention. For a better understanding of the invention with advantagesand features, refer to the description and to the tables and drawings.

BRIEF DESCRIPTION OF THE TABLES AND DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying Tables anddrawings in which:

Table 1 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row

Table 2 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row.

Table 3 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row.

Table 4 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row in columns 1 through 4after scrubbing all addresses has completed.

FIG. 1 shows the steps for beginning the preferred embodiment method 1for scrubbing central storage in a non-linear fashion favoring the CPAregion.

FIG. 2 shows the steps for beginning the second preferred embodimentmethod 2 for central storage.

FIG. 3 shows the steps taking the output of FIG. 2 and scrubbing the CPArows, with additional steps that can be taken.

FIG. 4 shows the steps used in scrubbing successively all chip rows toreturn to a ready state for starting the preferred embodiment processbegun in the steps shown in FIG. 1.

The detailed description explains the preferred embodiments of theinvention, together with advantages and features, by way of example withreference to the drawings.

DETAILED DESCRIPTION OF THE INVENTION

In accordance with our preferred embodiment, we have achieved ourimprovements by giving scrubbing preference to the central storageregion having the characteristic of a high risk read-only memory andthis prevents the accumulation of temporary data errors. The controlprogram area (CPA) is such a region in central storage given preference.The chip row on which the CPA resides is scrubbed after each time thescrubbing of a non-CPA chip row in a PMA completed successfully. Thenext non-CPA least recently scrubbed chip row is selected for scrubbingafter scrubbing completed on the CPA chip row.

Further, by adding hardware monitor indicators, for each chip rowconfigured in central storage, which monitor and are connected to thecustomary data error detection logic where the data error detectionlogic monitors all fetch accesses to central storage. There are assignedspecifically two sets of indicators (one CE hold latch and 1 UE holdlatch per set) for each chip row. These indicators are used to furtheridentify regions of central storage needing greater scrubbing focus inan effort to prevent a UE occurrence by correcting a temporary dataerror or by sparing a faulty DRAM before a system access is made fromwhich the system might not be able to recover. The first set ofindicators for each chip row would hold the current status and thesecond set would hold the previous status with the second set taking onthe value of the first set when all locations within the current chiprow being scrubbed have been completed.

Any time a CE or UE is detected, the appropriate CE or UE indicator forthe chip row associated with the address of the ECC word with the errorwould be set. When scrubbing has been completed for a chip row and thenext chip row is to be selected for scrubbing, the previous and currentchip row indicators for all chips rows, except the chip row wherescrubbing just finished, are examined for a new data error conditionindicating needed scrubbing. That is, should the previous indicators fora chip row have been CE=0 and UE=0 and the current indicators be CE=1 orUE=1, that would indicate a need for scrubbing. In addition to a chiprow needing scrubbing, the need would be more severe in the case of theUE relative to the CE. The severity of the detected data errordetermines the next scrub region.

In accordance with our method, the severity of the detected data erroralso determines the scrub speed. The selected chip row is scrubbed morequickly when the need is more severe than if the need was less severe.Four scrub speeds are provided to cover the four severities inincreasing severity order of 1) no new data error condition —normalspeed, 2) new CE detected—high speed, 3) new UE detected—higher speed,and 4) Half Spare mode—highest speed.

Turning now to our inventions in greater detail, it should be understoodthat the capabilities of the present invention can be implemented insoftware and or firmware in combination with server hardware.

For servers of our preferred embodiment as illustrated by the Figures,scrubbing favor (method 1 is our first preferred embodiment) asillustrated in the steps of FIG. 1 would be given to the region ofcentral storage having the characteristic of a high risk read-onlymemory, because of the potential accumulation of temporary errors. Thecontrol program area determined the area for scrubbing in FIG. 1,preferably the CPA region as this region would have this type ofcharacteristic. The chip row on which the CPA resides would be scrubbedafter each time the scrubbing of a non-CPA chip row in a PMA completedsuccessfully. The next non-CPA least recently scrubbed chip row would beselected for scrubbing after scrubbing completed on the CPA chip row.The chip row order would be CPA (normally chip row 0), chip row 1, CPA,chip row 2, CPA, chip row 3, CPA, chip row 1, etc. for regular scrubbingwith no DRAM sparing. The entire memory would be scrubbed in 16.2 hourswith the CPA region being scrubbed every 1.08 hours. This represents, itwill be noted, a significant advance.

When it has been determined that DRAM sparing is required at thecompletion of scrubbing a chip row because a Bit Error Counter reachedthreshold, the same chip row will be re-scrubbed in Half-Spare mode tomove data from the faulty DRAM to the spare DRAM. Because data are stillbeing fetched from the faulty DRAM during this time, the scrubbing rateis increased to the highest speed to complete the data move as quicklyas possible to minimize the system exposure to a potential UE. After thechip row has been re-scrubbed in Half Spare mode, the mode is set toFull Spare mode with future fetch accesses coming from the spare DRAM.As an example for a computer system with n chip rows and a sparingevent, the chip row order would be CPA, 1, CPA, 2, 2, CPA, n, CPA, 1,etc. with the re-scrubbing of chip row 2 after sparing a DRAM in chiprow 2.

Method 1 representing the preferred embodiment can be altered to provideeven more coverage over time by scrubbing the CPA chip row multipletimes before scrubbing a non-CPA chip row. For example, to provide 90%scrub coverage over time of the CPA area, the chip row order can be CPA,CPA, CPA, CPA, CPA, CPA, CPA, CPA, CPA, 1, CPA, CPA, CPA, CPA, CPA, CPA,CPA, CPA, CPA, 2, CPA (9 times), n, CPA (9 times), 1, etc.

As illustrated by FIG. 2, central storage memory controllers have fetchCE and UE detection logic that is used for normal system operations aswell as scrub operations. The detection logic monitors the data beingreturned from central storage.

For method 2 (our second preferred embodiment) on computer systems withadded hardware monitor indicators, there would be 2 sets of indicators(one CE hold latch and 1 UE hold latch per set), connected to thedetection logic, for each chip row configured in central storage. Theseindicators would be used to further identify regions of central storageneeding greater scrubbing focus in an effort to prevent a UE occurrenceby correcting a temporary CE or by sparing a faulty DRAM before a systemaccess is made from which the system might not be able to recover. Thefirst set of indicators for each chip row would hold the current statusand the second set would hold the previous status with the second settaking on the value of the first set when all locations within thecurrent chip row being scrubbed were completed. At the same time thefirst set for the completed chip row would be reset.

Any time a CE or UE is detected, the appropriate CE or UE indicator forthe chip row associated with the address of the ECC word with the errorwould be set. When scrubbing has been completed for a chip row and thenext chip row is to be selected for scrubbing, the previous and currentchip row indicators for all chips rows, except the chip row wherescrubbing just finished, would be examined for a new data errorcondition. That is, as an example, should the previous indicators for achip row have been CE=0 and UE=0 and the current indicators be CE=1 orUE=1, that would indicate a need for scrubbing. In addition to a chiprow needing scrubbing, the need would be more severe in the case of theUE relative to the CE and a CE relative to no error. The severity of thedetected data error determines the next scrub region.

As illustrated by FIG. 3, the severity of the detected data error alsodetermines the scrub speed. The selected chip row is scrubbed morequickly when the need is more severe than if the need was less severe.Four scrub speeds would cover the 4 severities in increasing severityorder of 1) no new data error condition—normal speed, 2) new CEdetected—high speed, 3) new UE detected—higher speed, and 4) Half Sparemode—highest speed.

Refer to the tables in the following section for the complete definitionof the selection logic and the next state logic.

Table 1 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row. A ‘1’ in column 5 pointsto a set of status conditions where a new UE has been recorded wherepreviously none existed. TABLE 1 Current Previous Previous UE Status UEStatus Current CE Status CE Status New UE Detected 0 0 0 0 0 0 0 0 1 0 00 1 0 0 0 0 1 1 0 0 1 0 0 0 0 1 0 1 0 0 1 1 0 0 0 1 1 1 0 1 0 0 0 1 1 00 1 1 1 0 1 0 1 1 0 1 1 1 1 1 0 0 0 1 1 0 1 0 1 1 1 0 0 1 1 1 1 0

Table 2 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row. A ‘1’ in column 5 pointsto a set of status conditions where a new CE has been recorded wherepreviously none existed and no new UE has been recorded. TABLE 2 CurrentPrevious Previous New CE Detected UE Status UE Status Current CE StatusCE Status w/no New UE 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 1 1 0 0 1 0 0 00 1 0 1 0 0 1 1 0 1 0 1 1 1 0 1 0 0 0 0 1 0 0 1 0 1 0 1 0 0 1 0 1 1 0 11 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 1 0

Table 3 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row. A ‘1’ in column 5 pointsto a set of status conditions where there is no new UE and no new CErecorded. TABLE 3 Static Condition Current Previous Previous No New UEUE Status UE Status Current CE Status CE Status No New CE 0 0 0 0 1 0 00 1 1 0 0 1 0 0 0 0 1 1 1 0 1 0 0 1 0 1 0 1 1 0 1 1 0 0 0 1 1 1 1 1 0 00 0 1 0 0 1 0 1 0 1 0 0 1 0 1 1 0 1 1 0 0 1 1 1 0 1 1 1 1 1 0 0 1 1 1 11

Table 4 illustrates the 16 possible combinations for the current andprevious UE and CE indicators for a chip row in columns 1 through 4after scrubbing all addresses has completed. The current status incolumns 1 and 3 are pushed (moved) into the status indicators holdingthe next state of the previous status indicators as seen in columns 7and 9 with the next state of the current status indicators now beingreset as seen in columns 6 and 8. TABLE 4 Previous Previous Next state -Next state - Next state - Next state - Current UE UE Current CE UECurrent UE Previous Current CE Previous indicator indicator indicatorindicator indicator UE indicator CE after scrub after scrub after scrubafter scrub after push indicator after push indicator completescompletes completes completes down and after push down and after pushchip row chip row chip row chip row reset down reset down 0 0 0 0 0 0 00 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 01 0 0 0 0 0 1 1 0 0 0 0 1 0 1 1 1 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 1 0 1 00 1 0 1 0 0 1 0 1 1 0 1 1 0 1 0 1 1 1 0 0 0 1 0 0 1 1 0 1 0 1 0 0 1 1 10 0 1 0 1 1 1 1 1 0 1 0 1

Further details of the flow utilized by the methods of the preferredembodiments are specified explicitly by the steps set forth in the FIGS.1, 2, 3 and 4. A review of the detailed steps there shown is selfexplanatory. These steps can be controlled by software, microcode orfirmware in combination with the server's hardware.

While the preferred embodiment to the invention has been described, itwill be understood that those skilled in the art, both now and in thefuture, may make various improvements and enhancements which fall withinthe scope of the claims which follow. The flow diagrams depicted hereinare examples of the preferred embodiment. There may be many variationsto these diagrams or the steps (or operations) described therein withoutdeparting from the spirit of the invention. For instance, the steps maybe performed in a differing order, or steps may be added, deleted ormodified consistent with the detailed description herein, as may occurto those skilled in the art both now and in the future. All of thesevariations are considered a part of the claimed invention. These claimsshould be construed to maintain the proper protection for the inventionfirst described.

1. A method for scrubbing regions in central storage of a computersystem, comprising the steps of: successively scrubbing newly selectedstorage chip rows of a Processor Memory Array (PMA) and proceeding firstwith selection for scrubbing preference given to scrubbing a centralstorage region having the characteristic selected by a nonlinearselection of the scrub region to be first scrubbed.
 2. The methodaccording to claim 1 wherein the nonlinear selection criteria determinesa characteristic region to be first scrubbed is a high risk read-onlymemory region and the process proceeds to scrub chip rows successivelyfirst in the selected region given scrubbing preference to prevent theaccumulation of temporary data errors to scrub the Processor MemoryArray.
 3. The method according to claim 1 wherein the region of selectedpreference is determined to be a control program area (CPA) for thepreferred the preferred central storage region.
 4. The method accordingto claim 3 wherein the chip row on which the CPA resides is scrubbedafter each time the scrubbing of a non-CPA chip row in a ProcessorMemory Array (PMA) completed successfully.
 5. The method according toclaim 4 wherein after scrubbing the CPA chip row on which the CPAresides, the next non-CPA least recently scrubbed chip row is selectedfor scrubbing.
 6. The method according to claim 1 wherein, with hardwaremonitor indicators for each chip row configured in central storageconnected to the customary data error detection logic said data errordetection logic monitors all fetch accesses to central storage.
 7. Themethod according to claim 6 wherein there are two sets of hardwaremonitor indicators (one CE hold latch and 1 UE hold latch per set) foreach chip row.
 8. The method according to claim 6 wherein there are setsof hardware monitor indicators for each row, each set having a CE holdlatch and a UE hold latch.
 9. The method according to claim 8 whereinsaid hardware monitor indicators are used to further identify regions ofcentral storage needing greater scrubbing focus in an effort to preventa UE occurrence by correcting a temporary data error or by sparing afaulty DRAM of said Processor Memory Array (PMA) before a system accessis made from which the system might not be able to recover.
 10. Themethod according to claim 7 wherein a first of said set of hardwaremonitor indicators for each chip row would hold the current status and asecond set would hold the previous status with the second set taking onthe value of the first set when all locations within the current chiprow being scrubbed is completed.
 11. The method according to claim 8wherein any time a CE or UE is detected for an ECC word with an errorthen an appropriate CE or UE hardware monitor indicator for the chip rowassociated with the address of the ECC word with the error will be set.12. The method according to claim 11 wherein when scrubbing has beencompleted for a chip row and the next chip row is to be selected forscrubbing, the previous and current chip row indicators for all chipsrows, except the chip row where scrubbing just finished, aresuccessively examined for a new data error condition.
 13. The methodaccording to claim 12 wherein should the previous indicators for a chiprow have been CE=0 and UE=0 and the current indicators be CE=1 or UE=1,such values indicate a need for scrubbing.
 14. The method according toclaim 13, wherein, in addition to a chip row needing scrubbing, a UEerror would be favored for selection for first scrubbing over a case ofUE error condition of equal severity.
 15. The method according to claim1 wherein hardware monitor indicators indicate a level of severity of adetected data error and the severity of the detected data errordetermines the next scrub region.
 16. The method according to claim 15wherein the severity of the detected data error also determines thescrub speed.
 17. The method according to claim 16 wherein the selectedchip row is scrubbed more quickly when the need is more severe than ifthe need were less severe.
 18. The method according to claim 17 whereinfour scrub speeds may indicate four severities in increasing severityorder: (1) no new data error condition—normal speed; (2) new CEdetected—high speed; (3) new UE detected—higher speed; and (4) HalfSpare mode—highest speed.